Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 313265 reports (Page 41 of 6266)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 773.295853ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 15:57:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 773.333312ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.028606523s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 15:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 2.028624573s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.788908827s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:57:51 UTC) orbiter warning: unexpected EOF Duration: 1.788957962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.39188775s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 15:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.391918168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.008178648s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 15:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 2.008206271s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.410878645s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 15:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1.410900485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.098353443s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:57:20 UTC) orbiter warning: unexpected EOF Duration: 2.098377929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.813937637s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 15:56:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 1.813963293s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.854650878s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 15:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 2.854683956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 668.609147ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 15:56:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 668.637081ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 12.276656857s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 15:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 12.276679399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 15.743294997s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:56:27 UTC) orbiter warning: unexpected EOF Duration: 15.743325345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002644563s trace: https://jaeger.dev.storj.io/trace/5f119de0e448faf7 FAIL (2026-04-03 15:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece... Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments... Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil> Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments... Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil> Duration: 1m0.00266952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 972.972896ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 15:56:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 973.022402ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.860305723s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 15:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 2.860334184s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 11.718792866s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:55:51 UTC) orbiter warning: unexpected EOF Duration: 11.718812747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 24.314583313s trace: https://jaeger.dev.storj.io/trace/4f967f0a45df067b FAIL (2026-04-03 15:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments... Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment... Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment. Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece... Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments... Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil> Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments... Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil> Duration: 24.314649714s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 52.818672124s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:55:40 UTC) orbiter warning: unexpected EOF Duration: 52.818712857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 895.274159ms trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 15:55:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 895.320388ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001579421s trace: https://jaeger.dev.storj.io/trace/757734936509437c FAIL (2026-04-03 15:54:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001604833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 777.426239ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:53:37 UTC) orbiter warning: unexpected EOF Duration: 777.458859ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 795.643299ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:53:10 UTC) orbiter warning: unexpected EOF Duration: 795.723951ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 718.477872ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:52:20 UTC) orbiter warning: unexpected EOF Duration: 718.526469ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.064873224s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:51:54 UTC) orbiter warning: unexpected EOF Duration: 1.064976259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 782.397629ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:51:05 UTC) orbiter warning: unexpected EOF Duration: 782.437051ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 788.327223ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:50:36 UTC) orbiter warning: unexpected EOF Duration: 788.445021ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 928.148528ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:49:41 UTC) orbiter warning: unexpected EOF Duration: 928.198722ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 630.132649ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:48:57 UTC) orbiter warning: unexpected EOF Duration: 630.164525ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 614.449205ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:48:27 UTC) orbiter warning: unexpected EOF Duration: 615.249125ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 907.447164ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:48:06 UTC) orbiter warning: unexpected EOF Duration: 907.754683ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 842.340755ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:47:41 UTC) orbiter warning: unexpected EOF Duration: 842.409057ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 921.246395ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:47:17 UTC) orbiter warning: unexpected EOF Duration: 921.280072ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 598.275634ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:46:24 UTC) orbiter warning: unexpected EOF Duration: 598.304279ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 726.497957ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:45:47 UTC) orbiter warning: unexpected EOF Duration: 726.540203ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 759.507799ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:45:07 UTC) orbiter warning: unexpected EOF Duration: 759.606679ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.022361284s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:43:54 UTC) orbiter warning: unexpected EOF Duration: 1.022389028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 668.7589ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:43:30 UTC) orbiter warning: unexpected EOF Duration: 668.798474ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 707.066718ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:42:51 UTC) orbiter warning: unexpected EOF Duration: 707.133074ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 951.944492ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:42:30 UTC) orbiter warning: unexpected EOF Duration: 951.994876ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 724.487366ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:42:07 UTC) orbiter warning: unexpected EOF Duration: 724.522778ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.239579503s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:40:55 UTC) orbiter warning: unexpected EOF Duration: 1.239608814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.234086727s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:40:08 UTC) orbiter warning: unexpected EOF Duration: 1.234128662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.073828456s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:39:14 UTC) orbiter warning: unexpected EOF Duration: 1.073880145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 712.861847ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:37:59 UTC) orbiter warning: unexpected EOF Duration: 712.895176ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.082712868s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:37:22 UTC) orbiter warning: unexpected EOF Duration: 1.082802607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 795.561497ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:36:07 UTC) orbiter warning: unexpected EOF Duration: 795.592934ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 782.779237ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:35:28 UTC) orbiter warning: unexpected EOF Duration: 782.832686ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 808.26769ms trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:34:13 UTC) orbiter warning: unexpected EOF Duration: 808.301572ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.093849235s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:33:04 UTC) orbiter warning: unexpected EOF Duration: 1.093957201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.022997476s trace: https://jaeger.dev.storj.io/trace/14dacdb9d69baa96 FAIL (2026-04-03 15:32:42 UTC) orbiter warning: unexpected EOF Duration: 1.023049144s core uplink us-select-1 us1-placement-20

1
...
39
40
41
42
43
...
6266