Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
1.265423791s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:07:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.265455417s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
1.197292571s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:07:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.197326581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............................................X...............] duration:
523.535005ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:07:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 523.565272ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
684.7757ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:07:02 UTC)
orbiter warning: unexpected EOF
Duration: 684.807077ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.206914769s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:06:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.206947266s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.201144599s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:06:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.20117601s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
791.165732ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:06:28 UTC)
orbiter warning: unexpected EOF
Duration: 791.291327ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.11168707s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:06:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.111718993s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................................X..............] duration:
619.847602ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:06:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 619.883928ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................................X.............] duration:
658.070323ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:06:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 658.105654ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
948.55114ms trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:05:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 948.580447ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
837.095206ms trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:05:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 837.128058ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................................X............] duration:
461.446866ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:05:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 461.473515ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
735.058104ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:05:15 UTC)
orbiter warning: unexpected EOF
Duration: 735.097838ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.23025257s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:04:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.230349059s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.251767437s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:04:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.251802516s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
606.138557ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:04:18 UTC)
orbiter warning: unexpected EOF
Duration: 606.203191ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................................X...........] duration:
513.139497ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:04:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 513.171287ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.167982538s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:03:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.16800887s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
670.301043ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:03:50 UTC)
orbiter warning: unexpected EOF
Duration: 670.348293ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................................X..........] duration:
488.275581ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:03:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 488.304295ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
777.194942ms trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:03:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 777.222118ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.518392677s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:03:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.518486399s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.127203705s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:03:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.12725645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
691.301725ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:03:04 UTC)
orbiter warning: unexpected EOF
Duration: 691.41503ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
721.308845ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:02:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 721.342711ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.364151021s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:02:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.364247128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.132843939s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:02:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.132876019s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
562.283626ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:02:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 562.319002ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
765.754036ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:02:00 UTC)
orbiter warning: unexpected EOF
Duration: 765.783145ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.102060889s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:01:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.102097734s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
852.318676ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:01:39 UTC)
orbiter warning: unexpected EOF
Duration: 852.349473ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
1.268863008s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:01:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.26889597s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
552.2186ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 16:01:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 552.251157ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.178836258s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:00:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.17887216s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
745.734534ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:00:49 UTC)
orbiter warning: unexpected EOF
Duration: 745.76788ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.146885755s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:00:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.146911639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
1.210257121s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 16:00:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.210293176s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
732.890614ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 16:00:22 UTC)
orbiter warning: unexpected EOF
Duration: 732.932947ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.357504408s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 16:00:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.357596977s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
717.356489ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 15:59:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 717.388055ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
1.535709115s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 15:59:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.535740178s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.365685017s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 15:59:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 1.36577464s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
926.074558ms trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 15:59:10 UTC)
orbiter warning: unexpected EOF
Duration: 926.120344ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.532922717s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 15:58:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 1.532953365s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
1.260575576s trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 15:58:53 UTC)
orbiter warning: unexpected EOF
Duration: 1.260799188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
585.063826ms trace:
https://jaeger.dev.storj.io/trace/757734936509437c
FAIL (2026-04-03 15:58:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.108231ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.084135633s trace:
https://jaeger.dev.storj.io/trace/14dacdb9d69baa96
FAIL (2026-04-03 15:58:26 UTC)
orbiter warning: unexpected EOF
Duration: 2.084365707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.866323265s trace:
https://jaeger.dev.storj.io/trace/5f119de0e448faf7
FAIL (2026-04-03 15:58:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.847450416 (676h55m20.44422145s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.850484423 (676h55m20.447255466s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851265816 (676h55m20.448036848s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.851335271 (676h55m20.44810629s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.852632244 (676h55m20.449403276s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.853759523 (676h55m20.450530557s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.854308942 (676h55m20.451079974s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855598741 (676h55m20.452369775s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.855856612 (676h55m20.452627645s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.856878744 (676h55m20.453649825s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857363987 (676h55m20.454135022s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.857517507 (676h55m20.454288539s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.858364801 (676h55m20.455135835s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.859013040 (676h55m20.455784066s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.860270016 (676h55m20.457041049s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.861139660 (676h55m20.457910695s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Uploading piece...
Apr 3 15:55:22.946956079 (676h55m20.543727103s) [upload=758105]: Next returned nil segment
Apr 3 15:55:22.946978182 (676h55m20.543749207s) [upload=758105]: Waiting for error group managing segments...
Apr 3 15:55:22.950613762 (676h55m20.547384786s) [noise=true , address=77.90.31.12:4596 , piece_id=RQ5EXCYRDFZLPGRZOGZ6P7YVEVSZFJRUA4VX5N6FTSTAWGZPRACQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.953456627 (676h55m20.550227664s) [noise=true , address=145.79.227.2:4139 , piece_id=DBXYRUD4LNRSNUB24NNWELFDXYFBFJTLVZCBXYYODZGEWCGOJR5Q, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.985925871 (676h55m20.582696906s) [noise=true , address=45.91.116.10:4116 , piece_id=BYRYYY7LLNK64EQMYXB6VHBFYDNC7OUOZ6W3WFBUQ4TU5FEVCUMA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:22.991293230 (676h55m20.588064263s) [noise=true , address=77.90.31.19:4859 , piece_id=5VTBQ5DBIC4N63KC3F6BVPDPS3HEW2RYE3GZ3K73K7V232DIQRCA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.374477599 (676h55m20.971248623s) [noise=true , address=45.91.116.12:4247 , piece_id=3IESIPCO3XVSIJ33EEXW6FRZ3G37YVKZME6R35N2ZO3OEBI4SRLA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.410516329 (676h55m21.007287362s) [noise=true , address=45.91.116.5:4317 , piece_id=YXTRNSV5ECR2DUFZFUU4XVNF47ML4U3GAGTYCMYJ44IJKI4BZZUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.577138229 (676h55m21.173909253s) [noise=true , address=45.91.116.13:4183 , piece_id=H2LDXPXR2A3EPEUMETX44O22HTREGDNSWMBERVQ5WDPHERNZCRGA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.950875607 (676h55m21.547646622s) [noise=true , address=145.79.227.13:4809 , piece_id=7H6F5WOUWP7FBBMLV4R2YC2OVSUQLUTAYVJQKT3OXLYQZU4FVAQQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:23.988052242 (676h55m21.584823256s) [noise=true , address=45.91.116.9:4277 , piece_id=SX65IIZV42QQSU74OWQRKHSQGABEQ7HRDUGIPYGWA6GDPT2UVEWQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.337556551 (676h55m21.934327576s) [noise=true , address=145.79.227.6:4286 , piece_id=DITORKC5NAOUB563SP4SAE7EPNPEJPBTMJEKE7VSA3ZRLAHMF5KA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:24.554652696 (676h55m22.151423723s) [noise=true , address=193.149.164.71:4303 , piece_id=TQBJMOIRFYWFC3L4J4RWUXC622JVZO6N6KTPHSH4V3KH5UTLHCUQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: ecclient: upload failed (node:12VHFzWdrd4NbU6gEyP8umDftQusdLJryKF4B2RjzrNd1KD6RcA, address:193.149.164.71:4303): protocol: expected piece hash; hashstore: write /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/dd/log-00000000000011dd-00005043: input/output error; EOF
Apr 3 15:55:24.728599189 (676h55m22.325370231s) [noise=true , address=193.149.164.70:4118 , piece_id=KUVMXYT4AGGNLSFDYWQVWQHCTIS5UO6RWAXCANALK67J7X7HHJBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:25.907239271 (676h55m23.504010308s) [noise=true , address=77.90.31.17:4804 , piece_id=X27Y3ON6LOYAWAGK3HCPQPJI3BE33G377OQZYY7C6GXFDGNETLAA, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:26.561450821 (676h55m24.158221844s) [noise=true , address=145.79.227.33:4893 , piece_id=JMPKUOJFD77GEL6DE77C65JHU4FC6LYDEUFR2EDGLKUG4UCS3EHQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894520913 (676h55m25.491291966s) [noise=true , address=193.149.164.67:4013 , piece_id=BTNLTSQDYJV5FSL6FMHHR5LHRPGGHRB3WSWAN2PDFF5WRZS77EOQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: <nil>
Apr 3 15:55:27.894587262 (676h55m25.491358296s) [seg_pos={0 0}, upload=758105]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:27.894849922 (676h55m25.491620953s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fvEz2PQvJaFsBMCYJDo7QFTpoTNeBixWwN5SbwfrASDuWpwyn): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.894862658 (676h55m25.491633689s) [noise=true , address=193.149.164.68:4033 , piece_id=ONZCSUAEXJKOFBDW3HS6IFQCTWKNNPNKEM7YQJPNGEZF5NG6SWBQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895496151 (676h55m25.492267297s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12su9NwVVQHVh8qsDP2CQxDYQnKq4zBL3Qf8aYSYFtYCxVC7PDx): protocol: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58024->193.149.164.66:4981: use of closed network connection
Apr 3 15:55:27.895510422 (676h55m25.492281455s) [noise=true , address=193.149.164.66:4981 , piece_id=A3K2F2AZRQ2Y7U2RHM3EAENAV7SZL4JCZXJQZB77DFVAN7BQXHXQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895656755 (676h55m25.492427789s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwQmj7YDznDUJxgETxw7Xx2UxnGbW2P67ViXbVp9emjrMVB8k): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:27.895667238 (676h55m25.492438269s) [noise=true , address=193.149.164.69:4072 , piece_id=D7YJG6RO57AZ7LQXSG2VAFIZFUSQ4GKF65HHOGGQHOFD53DLNCGQ, seg_pos={0 0}, upload=758105]: Long tail cancellation due to optimal threshold
Apr 3 15:55:27.895690309 (676h55m25.492461333s) [seg_pos={0 0}, upload=758105]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:27.895788470 (676h55m25.492559501s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:27.895795115 (676h55m25.492566149s) [upload=758105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:27.942538597 (676h55m25.539309623s) [upload=758105]: Done uploading segments...
Apr 3 15:55:27.942558390 (676h55m25.539329413s) [upload=758105]: Upload finished. err: <nil>
Duration: 2.866356701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.000470649s trace:
https://jaeger.dev.storj.io/trace/4f967f0a45df067b
FAIL (2026-04-03 15:58:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Apr 3 15:55:20.264902369 (676h55m17.861673399s) [upload=758104]: Starting upload
Apr 3 15:55:20.265001397 (676h55m17.861772414s) [upload=758104]: Done starting upload
Apr 3 15:55:20.265235236 (676h55m17.86200625s) [upload=758104]: Uploading segments...
Apr 3 15:55:20.265360138 (676h55m17.862131163s) [upload=758104]: Got next segment. Inline: false
Apr 3 15:55:20.265453890 (676h55m17.862224919s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 3 15:55:20.265462469 (676h55m17.862233496s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 3 15:55:20.293920151 (676h55m17.890691184s) [seg_pos={0 0}, upload=758104]: Begin upload segment...
Apr 3 15:55:20.294007959 (676h55m17.890778989s) [seg_pos={0 0}, upload=758104]: Done begin upload segment.
Apr 3 15:55:20.294055432 (676h55m17.890826465s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294073409 (676h55m17.890844433s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294156300 (676h55m17.890927333s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.294650969 (676h55m17.891421999s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.295234432 (676h55m17.892005442s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.297724273 (676h55m17.894495306s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298450908 (676h55m17.895221943s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.298475495 (676h55m17.895246525s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299228345 (676h55m17.895999361s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.299495439 (676h55m17.896266478s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.300671147 (676h55m17.897442177s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.301156109 (676h55m17.897927127s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302089713 (676h55m17.898860726s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.302302074 (676h55m17.899073108s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303225102 (676h55m17.899996135s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.303838945 (676h55m17.900609978s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.306200311 (676h55m17.902971347s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.307740630 (676h55m17.904511664s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Uploading piece...
Apr 3 15:55:20.397795764 (676h55m17.994566777s) [upload=758104]: Next returned nil segment
Apr 3 15:55:20.397819995 (676h55m17.994591099s) [upload=758104]: Waiting for error group managing segments...
Apr 3 15:55:20.406561480 (676h55m18.0033325s) [noise=true , address=145.79.227.52:4478 , piece_id=L62IQ6IIFVYT2I2CRP3L4V5SEFEPOTT5YMCYS7UQS6PJBHUMUATQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.432908583 (676h55m18.029679608s) [noise=true , address=45.91.116.11:4150 , piece_id=4RZQ2BIXGM2BN7WXYTQA76UV447GFAYCXQNID7JII4NFSKBWD6IA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.439163617 (676h55m18.03593463s) [noise=true , address=77.90.31.18:4815 , piece_id=AM2EQV4V4KZ7QVSCKIZJQ5ACFX667Q22RYW6HWEFIDDXULCWCP3Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.463322196 (676h55m18.06009322s) [noise=true , address=45.91.116.14:4219 , piece_id=OLQ2GF2WOPCAVAAA4CKENJFYB4N6YU6WZHY3WVTIOMHB6LYPO7PA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.673147945 (676h55m18.269918971s) [noise=true , address=45.91.116.6:4980 , piece_id=7O7A3TIUEP4R4ZHSKCSBGMK25HMGTSEI4XVT6WZTQBGZ5RLM5Z4Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.908054113 (676h55m18.504825143s) [noise=true , address=64.49.13.27:4959 , piece_id=TNVQCQSO4CXDDI5MS57CEHP6Y33NOIAA2DPZSMRE26BZ4C24VMWQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:20.924845024 (676h55m18.521616049s) [noise=true , address=64.49.13.154:4931 , piece_id=MONFAVIVEG3ADRMY67EEBBT4K7NZQ4NXYOP753XTXSX7CDOPWU5A, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.269610919 (676h55m18.866381954s) [noise=true , address=145.79.227.6:4286 , piece_id=EKSGCHYZF5GBE3U6KFEZ7QAJEBRG2UIX3YG64XZKA4CIBT7WM7RA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.608309747 (676h55m19.20508078s) [noise=true , address=45.91.116.12:4237 , piece_id=MLK5QBJAG2QHQ2YJGSRY7HRAY26P3MWBDOOOFJM5ABFALRQQP74Q, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:21.958725133 (676h55m19.555496162s) [noise=true , address=145.79.227.13:4827 , piece_id=M6FPOMLMJGVEYIWTZW2VUQJAKGFIT54Z73YL4H5NB73TP7Z3HRIQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:22.573041319 (676h55m20.169812354s) [noise=true , address=193.149.164.71:4312 , piece_id=NHXWSLI7K57UWC2H2VQOYQCABO6ZDDRTIOQQJV7GDOLTRD6DKTGQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.081204610 (676h55m20.67797564s) [noise=true , address=193.149.164.69:4091 , piece_id=RWKYA73UDMOB6AEV5ET4RMPJEIXHFZ77VL7GQ2LUQG2CRFCOEQKA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.269016973 (676h55m20.865787999s) [noise=true , address=193.149.164.143:4702 , piece_id=OFBT5CIUITS7VMREZYKHQNG65UUGVVAGF65FTPST2BHH36VA5EAA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310889191 (676h55m20.907660214s) [noise=true , address=193.149.164.70:4135 , piece_id=KQ5LSA2LHSFLPLZOBFB24V2O4QV5P62RWJMDJP77CJ5JOJ5ZFFOA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: <nil>
Apr 3 15:55:23.310913182 (676h55m20.907684208s) [seg_pos={0 0}, upload=758104]: Satisfied cohort requirements after 14 pieces
Apr 3 15:55:23.311220798 (676h55m20.907991821s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SDh6Vg8YaU1vCemSdzbZyRanb47z8FyEWTVRCTMnp8KDa5Xh): protocol: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57234->193.149.164.68:4043: use of closed network connection
Apr 3 15:55:23.311252642 (676h55m20.908023667s) [noise=true , address=193.149.164.68:4043 , piece_id=KZOK55MDIHHAGIFDPGULONUI66WM4MSCC6LGF2TR2TE6HVJP5NMA, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.311818112 (676h55m20.908589146s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSyneZu6pBZn645y5xriDzXoaDSw4qfeWfZBitQzgqiPJ3hPw): protocol: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46824->193.149.164.67:4018: use of closed network connection
Apr 3 15:55:23.311833759 (676h55m20.908604793s) [noise=true , address=193.149.164.67:4018 , piece_id=ZOYB75YLWGYJXRHJO6QLVEBPHEZTIGCBOKZJUNMPO6HREIELH6JQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312125072 (676h55m20.908896105s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SyJ7QDRQNhnppJjvJ7rYeRfpGA8GZ5C1DVM91d3EzeQUCNxk8): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312136762 (676h55m20.908907788s) [noise=true , address=145.79.227.20:4446 , piece_id=AEEQHZF74KJBMDVM6YCYGSRR2N2UAC3EEH6XCUYO6DQOERVZN6NQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312489461 (676h55m20.909260488s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Apr 3 15:55:23.312502970 (676h55m20.909273997s) [noise=true , address=145.79.227.33:4881 , piece_id=KCERAGVJ5QAUAWGTWHWOHDICDHNQ2LPJPGORV2I4K7LMCP4LM5SQ, seg_pos={0 0}, upload=758104]: Long tail cancellation due to optimal threshold
Apr 3 15:55:23.312535767 (676h55m20.909306806s) [seg_pos={0 0}, upload=758104]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Apr 3 15:55:23.312672599 (676h55m20.909443622s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Apr 3 15:55:23.312678840 (676h55m20.909449864s) [upload=758104]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Apr 3 15:55:23.386235679 (676h55m20.983006704s) [upload=758104]: Done uploading segments...
Apr 3 15:55:23.386260865 (676h55m20.983031888s) [upload=758104]: Upload finished. err: <nil>
Duration: 2.000501311s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|