Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.....X...X....................X.........] duration: 445.330254ms FAIL (2025-07-09 02:31:27 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 445.357077ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 28.669629212s FAIL (2025-07-09 02:31:14 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 28.669661023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.....X...X....................X........] duration: 499.546897ms FAIL (2025-07-09 02:31:09 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 499.570834ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X....X....X.........X....X.X..XX.XX..X.X.X.....XXX.XX.X] duration: 30.001093255s FAIL (2025-07-09 02:31:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001135048s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.....X...X....................X.......] duration: 18.146523626s FAIL (2025-07-09 02:30:51 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 18.146561569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [X.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX..] duration: 41.88002185s FAIL (2025-07-09 02:30:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.880087909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.....X...X....................X......] duration: 9.018604356s FAIL (2025-07-09 02:30:22 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 9.018638055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 23.57223746s FAIL (2025-07-09 02:30:14 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 23.572274026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.....X...X....................X.....] duration: 461.814685ms FAIL (2025-07-09 02:29:53 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 461.995482ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.] duration: 59.802325251s FAIL (2025-07-09 02:29:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.802342393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 31.220476806s FAIL (2025-07-09 02:29:49 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 31.220575793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X....X....X.........X....X.X..XX.XX..X.X.X.....XXX.XX.] duration: 10.415326464s FAIL (2025-07-09 02:29:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.415341739s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......X....X....X.........X....X.X..XX.XX..X.X.X.....XXX.XX] duration: 30.001581459s FAIL (2025-07-09 02:29:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001617248s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.....X...X....................X....] duration: 479.629036ms FAIL (2025-07-09 02:29:22 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 479.655569ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 30.012436388s FAIL (2025-07-09 02:29:18 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 30.012484808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X....X....X.........X....X.X..XX.XX..X.X.X.....XXX.X] duration: 30.001375823s FAIL (2025-07-09 02:29:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001397812s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [.XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX] duration: 1m0.001348619s FAIL (2025-07-09 02:28:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001362191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.....X...X....................X...] duration: 443.485373ms FAIL (2025-07-09 02:28:37 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 443.510363ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 26.680311685s FAIL (2025-07-09 02:28:21 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 26.680338214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........X....X....X.........X....X.X..XX.XX..X.X.X.....XXX.] duration: 946.976318ms FAIL (2025-07-09 02:28:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 946.997529ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.....X...X....................X..] duration: 854.409148ms FAIL (2025-07-09 02:27:53 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 854.454418ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [..XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.X] duration: 1m0.001156142s FAIL (2025-07-09 02:27:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001198051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........X....X....X.........X....X.X..XX.XX..X.X.X.....XXX] duration: 30.001044377s FAIL (2025-07-09 02:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001062585s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........X....X....X.........X....X.X..XX.XX..X.X.X.....XX] duration: 30.001508622s FAIL (2025-07-09 02:27:14 UTC) orbiter warning: context deadline exceeded Duration: 30.001529867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..........................X.....X...X....................X.] duration: 431.518374ms FAIL (2025-07-09 02:27:11 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 431.557137ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 685.813528ms FAIL (2025-07-09 02:27:02 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 685.841737ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.] duration: 51.825155359s FAIL (2025-07-09 02:26:51 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.826001385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 34.428686821s FAIL (2025-07-09 02:26:31 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 34.428715133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..........................X.....X...X....................X] duration: 30.003225288s FAIL (2025-07-09 02:26:27 UTC) ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments... Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil> Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments... Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment... Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment. Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece... Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments... Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil> Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces. Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments... Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil> Duration: 30.003250079s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [............X....X....X.........X....X.X..XX.XX..X.X.X.....X] duration: 30.001083316s FAIL (2025-07-09 02:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001108737s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [X...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX] duration: 1m0.001196842s FAIL (2025-07-09 02:25:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00122183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 19.984763036s FAIL (2025-07-09 02:25:30 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 19.984794101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..........................X.....X...X....................] duration: 453.108578ms FAIL (2025-07-09 02:25:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.16083ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............X....X....X.........X....X.X..XX.XX..X.X.X.....] duration: 28.869556186s FAIL (2025-07-09 02:25:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.869627068s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXX] duration: 1m0.001261238s FAIL (2025-07-09 02:24:58 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001275671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..........................X.....X...X...................] duration: 560.55461ms FAIL (2025-07-09 02:24:52 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 560.582326ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 15.856163474s FAIL (2025-07-09 02:24:49 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 15.85619318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..........................X.....X...X..................] duration: 581.719578ms FAIL (2025-07-09 02:24:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 581.8272ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..............X....X....X.........X....X.X..XX.XX..X.X.X....] duration: 22.191385371s FAIL (2025-07-09 02:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.191409456s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXX] duration: 1m0.001739628s FAIL (2025-07-09 02:23:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001768528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............X....X....X.........X....X.X..XX.XX..X.X.X...] duration: 736.982367ms FAIL (2025-07-09 02:23:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 737.002834ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 30.643984453s FAIL (2025-07-09 02:23:51 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 30.6440139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..........................X.....X...X.................] duration: 537.212537ms FAIL (2025-07-09 02:23:45 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 537.308911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X..........................X.....X...X................] duration: 630.603498ms FAIL (2025-07-09 02:23:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 630.634312ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..XX...XX.XXX.XXX.X.X.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XX] duration: 1m0.000766416s FAIL (2025-07-09 02:22:57 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000783149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...............X....X....X.........X....X.X..XX.XX..X.X.X..] duration: 561.338254ms FAIL (2025-07-09 02:22:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 561.358077ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..........................X.....X...X...............] duration: 540.770849ms FAIL (2025-07-09 02:22:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.817759ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 17.138868409s FAIL (2025-07-09 02:22:28 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 17.138888954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..........................X.....X...X..............] duration: 530.567965ms FAIL (2025-07-09 02:22:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 6VoDjPavWXBNB34cestcm71): protocol: context canceled; context canceled Jul 9 02:08:42.004547501 (376h57m8.930842191s) [seg_pos={0 0}, upload=301887]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 02:08:42.004631001 (376h57m8.930925702s) [upload=301887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:08:42.004636167 (376h57m8.930930868s) [upload=301887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:08:42.053462218 (376h57m8.979756911s) [upload=301887]: Done uploading segments... Jul 9 02:08:42.053505095 (376h57m8.979799783s) [upload=301887]: Upload finished. err: <nil> Jul 9 02:09:51.570445507 (376h58m18.496740205s) [upload=301903]: Starting upload Jul 9 02:09:51.570597205 (376h58m18.496891895s) [upload=301903]: Done starting upload Jul 9 02:09:51.570974803 (376h58m18.497269477s) [upload=301903]: Uploading segments... Jul 9 02:09:51.571064500 (376h58m18.497359197s) [upload=301903]: Got next segment. Inline: false Jul 9 02:09:51.571112801 (376h58m18.497407498s) [upload=301903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:09:51.571118937 (376h58m18.497413636s) [upload=301903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:09:51.600053053 (376h58m18.52634774s) [seg_pos={0 0}, upload=301903]: Begin upload segment... Jul 9 02:09:51.600098515 (376h58m18.526393191s) [seg_pos={0 0}, upload=301903]: Done begin upload segment. Jul 9 02:09:51.600144175 (376h58m18.526438851s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600155495 (376h58m18.526450175s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600587878 (376h58m18.526882575s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600632093 (376h58m18.526926786s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.600942979 (376h58m18.527237673s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601022711 (376h58m18.527317409s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601286340 (376h58m18.52758102s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.601428018 (376h58m18.52772271s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603174680 (376h58m18.529469373s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603535278 (376h58m18.529829969s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.603850846 (376h58m18.530145536s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604148512 (376h58m18.530443203s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604476709 (376h58m18.530771401s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.604781059 (376h58m18.531075746s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605086191 (376h58m18.53138088s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.605432703 (376h58m18.531727398s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Uploading piece... Jul 9 02:09:51.694212643 (376h58m18.620507332s) [upload=301903]: Next returned nil segment Jul 9 02:09:51.694235136 (376h58m18.620529834s) [upload=301903]: Waiting for error group managing segments... Jul 9 02:09:51.697155194 (376h58m18.623449869s) [noise=true , address=77.90.31.13:4628 , piece_id=3XADEBLRXWZT6XR3OGYS3GNXD6J3XE3J3QERXMMB34BMGIER5XVA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.698887324 (376h58m18.625182015s) [noise=true , address=77.90.31.13:4627 , piece_id=NBJLVHTIMEZXNVY7EBCGGGHKPRRLYXRYGUK7UHPF5AE7FTIS6EAQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699500142 (376h58m18.625794854s) [noise=true , address=45.91.116.13:4154 , piece_id=FOTCXOFOSP6Z42NASI5DKOQB5NCKFOPGLU3AWEGIPZ64KG72RJCQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.699836015 (376h58m18.626130702s) [noise=true , address=45.91.116.11:4117 , piece_id=UFIZWESPQZSZFXHJYGVUS3KXVSEOAWPQWGFPF7V66QP62WBGKUVQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700095916 (376h58m18.626390609s) [noise=true , address=45.91.116.9:4262 , piece_id=JYZ4S77NQMOWBOLYMW2LPRYNKDBUVDRIDJUH3PC53RXD6W42ZTFA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.700470834 (376h58m18.626765522s) [noise=true , address=45.91.116.12:4226 , piece_id=MCPUTJKE7SXLXJUP2AMZ7436TZHEM7RZ54MHRZW5AQ3LNNFMAACQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703522403 (376h58m18.629817103s) [noise=true , address=104.36.82.9:4007 , piece_id=4NPIGVA3TX3VNUP6ZGUMTF2DVALOI7VB5JWS5ISRXSQ2G2UVC4EA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.703675296 (376h58m18.629969984s) [noise=true , address=104.36.82.16:4294 , piece_id=G3RZUWXIB236KBFFDDJHNB3C44YNKUAOJRBXCQQUH6LVNX3N7NTA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:51.959862838 (376h58m18.886157523s) [noise=true , address=45.91.116.14:4190 , piece_id=N4HV7NMMELJPVTYN5RYLXVCT43NMFL5AJSH5EXG4MKFQF4AFRZ6A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:09:52.136782791 (376h58m19.063077472s) [noise=true , address=45.91.116.14:4191 , piece_id=3GTKK2V5T6JATTQBWXQ53R2LHIF4DDJVVF4SM72SWFIT3EUCVJGQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:11.962323057 (376h58m38.888617733s) [noise=true , address=104.36.82.5:4839 , piece_id=HUJEMHQK4WS3YJT6XXUKX3J67FNSZYWHRBO2N5SCS3FBF63EPPJA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:14.610092164 (376h58m41.536386856s) [noise=true , address=104.36.82.10:4079 , piece_id=TCQTFYUSL435JSYVATD6Y3K2XRGCFML5HEZQCNEECJJPGMTYMTKQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: <nil> Jul 9 02:10:21.532800356 (376h58m48.459095046s) [noise=true , address=104.36.82.5:4837 , piece_id=PGSTFNKVTB6LUW3WZ34GQG26L45LORXMLT3PAWGWGJY3AWH3SG7A, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533507790 (376h58m48.459802485s) [noise=true , address=104.36.82.15:4257 , piece_id=YTJ4XIKP6PK5KOMJNGNI2QLUXT6NEKGD3K7C4VUH5JDEJJJTJRQQ, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 02:10:21.533692566 (376h58m48.45998726s) [noise=true , address=104.36.82.7:4911 , piece_id=BOEXOV5LOLEPRVFFZHODFM2LNS27MJK6J4PO6D4L3BWAGTSQEY7Q, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37038->104.36.82.7:4911: use of closed network connection Jul 9 02:10:21.533956295 (376h58m48.460250987s) [noise=true , address=104.36.82.11:4114 , piece_id=TVC6ZC6I7A3X4EB7W26Y4I4FJQQ5GLS6YO2JKNEHSJUYPLZZYWHA, seg_pos={0 0}, upload=301903]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:44176->104.36.82.11:4114: use of closed network connection Jul 9 02:10:21.533981601 (376h58m48.460276299s) [seg_pos={0 0}, upload=301903]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 02:10:21.575161371 (376h58m48.501456066s) [upload=301903]: Done uploading segments... Jul 9 02:10:21.575192579 (376h58m48.501487274s) [upload=301903]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 530.59782ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 19.490256099s FAIL (2025-07-09 02:22:11 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== 3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment... Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment. Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece... Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments... Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil> Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces. Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments... Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil> Duration: 19.490308171s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)