Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.188151646s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:33:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.188176939s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.163866837s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:33:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 2.163898916s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 698.419503ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:32:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 698.461563ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 749.222475ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 749.254617ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 967.101533ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:32:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 967.12707ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.103400314s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:31:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.103436805s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.211771091s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:31:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.211813699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.474728885s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:31:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474755086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.152579505s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.152611469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.270522193s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:30:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.270556154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.665693269s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:29:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.665748588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.321748955s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.321775584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 653.865021ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 653.893001ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.19428801s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:29:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.194320591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.327334969s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:28:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.327367892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.561556226s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.561581337s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.325846226s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:28:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.325893917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.125401197s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.125477629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.209844274s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:28:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.209885053s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 991.259821ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:28:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 991.293658ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.255159745s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:27:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.255188491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.158312225s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 4.158341752s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.876883374s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:26:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.87692465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.38369227s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:26:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.383734156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.018532615s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:26:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.018560588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.794869653s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 2.7949076s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.351918898s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 3.352095988s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.348952168s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:25:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.348985024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.221781955s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:25:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.221821437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 638.383317ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:24:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 638.418167ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.157724812s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.157762826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.37249293s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.372553104s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.222983928s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.223002529s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 463.428732ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:23:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 463.460792ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.143927068s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.143955376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.12487544s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:23:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.124908s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.085389722s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.085425889s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 718.992862ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:22:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 719.020007ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.240351904s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:22:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.240444563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 472.188395ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:22:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.355099ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 661.816063ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:21:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 661.838451ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 757.829641ms trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 757.859528ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 943.582406ms trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 943.616424ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 662.806925ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:20:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 662.991305ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 495.921811ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:20:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 495.956374ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.157161934s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.157350317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.223517293s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.223565769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 649.159039ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 16:20:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 649.189805ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.111677454s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 16:19:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.111705333s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.292074883s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 16:19:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.292121766s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)