Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
4.934335999s
FAIL (2025-07-09 02:54:05 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 4.934369045s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [.X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.XX.X..] duration:
8.928532444s
FAIL (2025-07-09 02:54:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.928555657s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
781.800163ms
FAIL (2025-07-09 02:53:43 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 781.825979ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [..X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.XX.X.] duration:
8.713718357s
FAIL (2025-07-09 02:53:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.713914032s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
400.70768ms
FAIL (2025-07-09 02:53:30 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 400.799484ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [...X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.XX.X] duration:
30.000672217s
FAIL (2025-07-09 02:53:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00070347s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [.XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..XXX.XX] duration:
1m0.001722081s
FAIL (2025-07-09 02:53:16 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001897323s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
9.951438863s
FAIL (2025-07-09 02:53:15 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 9.951471093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
544.612173ms
FAIL (2025-07-09 02:52:59 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 544.638435ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
21.249755304s
FAIL (2025-07-09 02:52:35 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 21.24977734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.XX.] duration:
3.501537267s
FAIL (2025-07-09 02:52:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.501566272s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
37 out of 60 is failed (threshold=3) [..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..XXX.X] duration:
1m0.002269198s
FAIL (2025-07-09 02:52:16 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.002313108s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X....................X......................................] duration:
447.082941ms
FAIL (2025-07-09 02:52:15 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 447.114577ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
37.805283159s
FAIL (2025-07-09 02:52:14 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 37.805302079s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X....................X.....................................] duration:
8.904118435s
FAIL (2025-07-09 02:52:04 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 8.904206785s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.XX] duration:
30.001232361s
FAIL (2025-07-09 02:51:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001759024s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
26.013342418s
FAIL (2025-07-09 02:51:36 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 26.013370748s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X....................X....................................] duration:
399.861184ms
FAIL (2025-07-09 02:51:34 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 399.892974ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
37 out of 60 is failed (threshold=3) [X..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..XXX.] duration:
45.848032312s
FAIL (2025-07-09 02:51:13 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 45.848060647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
20.163288303s
FAIL (2025-07-09 02:50:46 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 20.163317065s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X....................X...................................] duration:
10.561379941s
FAIL (2025-07-09 02:50:44 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 10.561491692s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.X] duration:
30.001742763s
FAIL (2025-07-09 02:50:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001770991s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [XX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..XXX] duration:
1m0.001240472s
FAIL (2025-07-09 02:50:23 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001290606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X...X....................X..................................] duration:
459.066732ms
FAIL (2025-07-09 02:49:47 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 459.093045ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
15.977768074s
FAIL (2025-07-09 02:49:33 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 15.977789637s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X.] duration:
8.70708406s
FAIL (2025-07-09 02:49:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.707111336s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [XXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..XX] duration:
1m0.001586641s
FAIL (2025-07-09 02:49:23 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001605366s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
21.338796771s
FAIL (2025-07-09 02:49:17 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 21.33882258s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [...X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..X] duration:
30.000702241s
FAIL (2025-07-09 02:49:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000723395s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X...X....................X.................................] duration:
575.382605ms
FAIL (2025-07-09 02:48:55 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 575.432957ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
22.97477073s
FAIL (2025-07-09 02:48:36 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 22.97481822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [XXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..X] duration:
1m0.000617932s
FAIL (2025-07-09 02:48:23 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000639869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [....X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X..] duration:
6.609833322s
FAIL (2025-07-09 02:48:13 UTC)
orbiter warning: context deadline exceeded
Duration: 6.609859903s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X...X....................X................................] duration:
830.505842ms
FAIL (2025-07-09 02:47:44 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 830.533713ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X.] duration:
596.354727ms
FAIL (2025-07-09 02:47:31 UTC)
orbiter warning: context deadline exceeded
Duration: 596.500808ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X...X....................X...............................] duration:
408.337393ms
FAIL (2025-07-09 02:47:27 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 408.367754ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
15.842797626s
FAIL (2025-07-09 02:47:21 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 15.842824915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
38 out of 60 is failed (threshold=3) [XXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX..] duration:
59.384700551s
FAIL (2025-07-09 02:47:19 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 59.384722553s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [......X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..X] duration:
30.001388393s
FAIL (2025-07-09 02:47:06 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001425595s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X...X....................X..............................] duration:
624.04312ms
FAIL (2025-07-09 02:46:43 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 624.076139ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
21.84788336s
FAIL (2025-07-09 02:46:27 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 21.847914494s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X...X....................X.............................] duration:
819.37522ms
FAIL (2025-07-09 02:46:16 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 819.405254ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
34.325254155s
FAIL (2025-07-09 02:46:04 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 34.325284121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.......X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X..] duration:
540.059734ms
FAIL (2025-07-09 02:45:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 540.104525ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX.] duration:
4.785240291s
FAIL (2025-07-09 02:45:22 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.785265044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
26.7564695s
FAIL (2025-07-09 02:45:18 UTC)
orbiter warning:
======ERROR======
uplink: bucket: metaclient: context deadline exceeded
======LOGS======
3:56.263910643 (377h2m23.19020535s) [upload=301954]: Got next segment. Inline: false
Jul 9 02:13:56.263954358 (377h2m23.190249034s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:13:56.263959891 (377h2m23.190254572s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:13:56.302204249 (377h2m23.228498944s) [seg_pos={0 0}, upload=301954]: Begin upload segment...
Jul 9 02:13:56.302291587 (377h2m23.22858628s) [seg_pos={0 0}, upload=301954]: Done begin upload segment.
Jul 9 02:13:56.302338947 (377h2m23.228633682s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302352599 (377h2m23.228647272s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302450837 (377h2m23.228745531s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.302958774 (377h2m23.229253667s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303069086 (377h2m23.229363782s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303495994 (377h2m23.22979069s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303575840 (377h2m23.229870536s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.303949062 (377h2m23.230245641s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304060410 (377h2m23.230355106s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304407993 (377h2m23.230702685s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304548115 (377h2m23.230842812s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.304795164 (377h2m23.231089862s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305012510 (377h2m23.231307211s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305113737 (377h2m23.231408427s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305383755 (377h2m23.231678434s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305483822 (377h2m23.231778517s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.305954019 (377h2m23.232248718s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.306425517 (377h2m23.232720213s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Uploading piece...
Jul 9 02:13:56.375470790 (377h2m23.301765481s) [upload=301954]: Next returned nil segment
Jul 9 02:13:56.375489934 (377h2m23.30178463s) [upload=301954]: Waiting for error group managing segments...
Jul 9 02:13:56.381456859 (377h2m23.307751539s) [noise=true , address=77.90.31.10:4525 , piece_id=GPVSIZIBZWTUCP5RFYVPFYIYIL2LWU5PU3UASQVRQLY7SSSFMQBA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.381956907 (377h2m23.308251586s) [noise=true , address=77.90.31.14:4689 , piece_id=LW7OT74AJCCC3TEKY6OZI553YK6YQFQ6C7PORQB53E7FUS24KQKQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.382837088 (377h2m23.309131761s) [noise=true , address=77.90.31.21:4898 , piece_id=75XXVNT4EV4D5UTX7O5XFL44IZOYY5LZSI2VPFQZPRERL6W5M23A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.393489125 (377h2m23.319783812s) [noise=true , address=77.90.31.21:4907 , piece_id=2EUWATHUQNZ77BJV65ZFKR5WL4D3AUURLBQ32XR2IVBA7P4TITOQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404692801 (377h2m23.330987481s) [noise=true , address=77.90.31.16:4739 , piece_id=H6UCV6RURPXKC2H3YGAT2OXV2VZMW2DWIRG6TFD6CVXKOVMDP2VQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.404986115 (377h2m23.331280795s) [noise=true , address=77.90.31.18:4838 , piece_id=TGWYXA3AF7LDG4KPVCRNEYFDADAG6TQ5DAX7JJVW5BLVVVNECILA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.433785086 (377h2m23.360079766s) [noise=true , address=77.90.31.13:4640 , piece_id=TPPN75DX3UCBM346CSVJN3GSOA47QV4FVPYSSAHWCJYEM42OCVLQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:13:56.598499747 (377h2m23.524794427s) [noise=true , address=79.127.201.215:4339 , piece_id=XDCTYE2SVI6RAPCXTRVANFCZYWJDZXLBHHJ47NQY6V7D36DEFPGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:11.107785584 (377h2m38.034080273s) [noise=true , address=104.36.82.22:4541 , piece_id=XN4HDFDNZ4YUJTDPOTMMDFX4FE2YVZDGCIMHEMJGWXRPNHQ6EU7A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.515567029 (377h2m39.441861713s) [noise=true , address=104.36.82.17:4345 , piece_id=YI27O5WLBYO3TPCX4MVUATFORY6U3EQMDKQHGKRHIZMTG7WGFB2A, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:12.923090587 (377h2m39.849385267s) [noise=true , address=104.36.82.8:4986 , piece_id=KPY5JVLSAEB5GUKUJJSIF5M6IIHTJISEYYAXCGUSJ7OLI5QCIBNQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:13.453172659 (377h2m40.379467346s) [noise=true , address=104.36.82.38:4698 , piece_id=TSOXJNQEWODCWFEITH4IYKZI7A75UZEXRQPIQO7LMP6LT3WLNXUA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.416524856 (377h2m41.342819534s) [noise=true , address=104.36.82.34:4554 , piece_id=57K5SDV3YEGLVMSJGSLXNKEWOLETXE6CWBM2NPVCNKLFHPVQCJBQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846730699 (377h2m41.773025379s) [noise=true , address=104.36.82.23:4464 , piece_id=JCJ4MGEZIIZ7YCTM6QXC5ZJ7PI4OEJBOMHZYBU57D2SNGWJWB5CQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: <nil>
Jul 9 02:14:14.846763777 (377h2m41.773058463s) [seg_pos={0 0}, upload=301954]: Segment reached optimal threshold of 14 pieces.
Jul 9 02:14:14.846894905 (377h2m41.773189582s) [noise=true , address=104.36.82.36:4644 , piece_id=K2G3MAPD7TWYZPKNHQBMEPJRIQDLE75FYVPAZFX4I3GJ4IUAA7XQ, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oYHbb73ACitbfkkP5P6cmRyTt6kmaS1y49G4vbsvTvVDwByhT): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.846948435 (377h2m41.773243131s) [noise=true , address=104.36.82.4:4834 , piece_id=YLYIASUPNRTK7KOMRYLN7OQLQBPWC7WFGCPDGUVIOEQJWQQ7A7GA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Z241DCYA5Wig2TdEogxTpY6FzVmFbnKtmHcTwxLxfcZafgooy): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847094561 (377h2m41.773389249s) [noise=true , address=104.36.82.42:4830 , piece_id=ENYQTUCIF7OWQPCXTYS24OC5QT63ZJJXOY652GLIG7OQPJEQCGGA, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9): protocol: expected piece hash; context canceled; EOF
Jul 9 02:14:14.847127825 (377h2m41.773422511s) [noise=true , address=104.36.82.41:4768 , piece_id=BGMF33OKUZSZKGIHJYI3RNXEBDE54TX67DYLJ7JCZMRNK6ZAU22Q, seg_pos={0 0}, upload=301954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43796->104.36.82.41:4768: use of closed network connection
Jul 9 02:14:14.847142999 (377h2m41.773437687s) [seg_pos={0 0}, upload=301954]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 02:14:14.847204962 (377h2m41.773499652s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:14:14.847209974 (377h2m41.773504671s) [upload=301954]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:14:14.909581364 (377h2m41.83587606s) [upload=301954]: Done uploading segments...
Jul 9 02:14:14.909605483 (377h2m41.835900172s) [upload=301954]: Upload finished. err: <nil>
Duration: 26.75662704s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
40 out of 60 is failed (threshold=3) [XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.XX] duration:
1m0.000688083s
FAIL (2025-07-09 02:45:17 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000702139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.....X...X....................X............................] duration:
404.523952ms
FAIL (2025-07-09 02:45:12 UTC)
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
02:25:27.135824454 (377h13m54.062119141s) [noise=true , address=104.36.82.12:4150 , piece_id=DRG5SBA3EE4KVQPEQOJZ6SS537SFWYFUYQXOAQMWYTT6FPVSJIPQ, seg_pos={0 0}, upload=302099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled
Jul 9 02:25:27.135860079 (377h13m54.062154769s) [seg_pos={0 0}, upload=302099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:25:27.135957283 (377h13m54.062251971s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:25:27.135964211 (377h13m54.062258897s) [upload=302099]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:25:27.199410713 (377h13m54.125705389s) [upload=302099]: Done uploading segments...
Jul 9 02:25:27.199427901 (377h13m54.125722577s) [upload=302099]: Upload finished. err: <nil>
Jul 9 02:25:57.289077655 (377h14m24.215372329s) [upload=302109]: Starting upload
Jul 9 02:25:57.289186750 (377h14m24.215481424s) [upload=302109]: Done starting upload
Jul 9 02:25:57.289444851 (377h14m24.215739548s) [upload=302109]: Uploading segments...
Jul 9 02:25:57.289623846 (377h14m24.215918535s) [upload=302109]: Got next segment. Inline: false
Jul 9 02:25:57.289696741 (377h14m24.215991432s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:25:57.289703375 (377h14m24.215998067s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:25:57.318313152 (377h14m24.244607838s) [seg_pos={0 0}, upload=302109]: Begin upload segment...
Jul 9 02:25:57.318363286 (377h14m24.244657964s) [seg_pos={0 0}, upload=302109]: Done begin upload segment.
Jul 9 02:25:57.318386772 (377h14m24.244681463s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318412127 (377h14m24.244706825s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318822149 (377h14m24.245116841s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.318965862 (377h14m24.24526056s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319187516 (377h14m24.245482203s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.319523815 (377h14m24.245818512s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.320556632 (377h14m24.24685133s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.321019141 (377h14m24.247313835s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.322591797 (377h14m24.248886493s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323120451 (377h14m24.249415138s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323544591 (377h14m24.249839288s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.323950189 (377h14m24.250244889s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.326585810 (377h14m24.252880507s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.327871214 (377h14m24.254165904s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.328605288 (377h14m24.254899986s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.335409087 (377h14m24.261703784s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Uploading piece...
Jul 9 02:25:57.475344637 (377h14m24.401639326s) [upload=302109]: Next returned nil segment
Jul 9 02:25:57.475376181 (377h14m24.401670879s) [upload=302109]: Waiting for error group managing segments...
Jul 9 02:25:57.477269278 (377h14m24.403563958s) [noise=true , address=45.91.116.7:4011 , piece_id=VDOJRN6ACK4LO44BQTULO6BB3CLAVQI7YIC6XLR32CZV4UJSPE6Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.477736179 (377h14m24.40403086s) [noise=true , address=45.91.116.9:4263 , piece_id=5YILAJ5BZBGOURSZE343ST4RMOTWM6ZOB3EEQQWL5WUNYWOGSAPA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478396652 (377h14m24.404691332s) [noise=true , address=77.90.31.10:4518 , piece_id=VJ4HWM2X4VSRSUCQL5QLMBYPFBYV4JYS5ZXCBCB6MTMB76EHZFGQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.478919538 (377h14m24.405214224s) [noise=true , address=45.91.116.14:4191 , piece_id=GTR6T4OTGZTEZ76FWQZ46OCH5WB3O5X27EN4AV33ZJ7D5HX4535A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481426716 (377h14m24.407721396s) [noise=true , address=45.91.116.4:4939 , piece_id=ANA3EZU7OZWELNG4QEMTEAYZY5H3C3H4IJFT4XKPUB7WIF44FEVQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481718373 (377h14m24.408013051s) [noise=true , address=45.91.116.5:4297 , piece_id=MSDWDLY5PJHQVI5565EKANJVSTMZFEUD2OTYZ52C65XQUICNMC3A, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.481930356 (377h14m24.408225033s) [noise=true , address=77.90.31.17:4770 , piece_id=V36UBBIYOFJC6KFVVBXEVT4QBUIWN257D3YSDDRAGZVXH7ZXHTBA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482376147 (377h14m24.408670821s) [noise=true , address=45.91.116.8:4045 , piece_id=KM6ZYTHR6JGTARKZQZYIESPKBEU5H2HQFA7OARIKD2ZXDHPFSMGA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.482439192 (377h14m24.408733868s) [noise=true , address=45.91.116.6:4973 , piece_id=Z7HQQ2O5QUSJKT4EMIDMEI5PIAEHNVRYX7MALUR2GQULRFIBUOKA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.483047970 (377h14m24.409342644s) [noise=true , address=77.90.31.13:4627 , piece_id=HULEKFVZNDNKS4SWWGTJUHYZWWYPDCER5W4GFCVLLDL65PAHULFQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:25:57.497788632 (377h14m24.424083312s) [noise=true , address=77.90.31.11:4553 , piece_id=TYW7NBJWACIT7OQS7A4WGXUPR3MB2FKAN2OMGAZJERB4X2DLAO2Q, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:16.688921171 (377h14m43.615215851s) [noise=true , address=104.36.82.6:4875 , piece_id=C275F2JPVAF52E34CEHT5QXCHUFGIQZ7XGMOAOUOT4RA7YOHRQLQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177893955 (377h14m54.104188635s) [noise=true , address=104.36.82.2:4732 , piece_id=DJIGHJMJRAULH2L4YT2WGRXPRQJMNKZG5LICHPSKOCHUL6QDV6BQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: <nil>
Jul 9 02:26:27.177985584 (377h14m54.104280281s) [seg_pos={0 0}, upload=302109]: Segment reached optimal threshold of 13 pieces.
Jul 9 02:26:27.178189789 (377h14m54.104484461s) [noise=true , address=104.36.82.5:4838 , piece_id=WK3ALBOEMWWTVEA4FKCS352SZ6YYB35LELURNWXQZK32K4HYDRNA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178252042 (377h14m54.104546733s) [noise=true , address=104.36.82.7:4909 , piece_id=ZRHLWDQ5KIDRWPJ5FXAANQTWERKXCR2PPEGN442IOSYSRFAG7VZA, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178273922 (377h14m54.104568608s) [noise=true , address=104.36.82.10:4078 , piece_id=ZFVUQAEV6DDFLPKCZA2XJ2BZHMMM745LGB4JRZJZQPG72WECGMDQ, seg_pos={0 0}, upload=302109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Jul 9 02:26:27.178305983 (377h14m54.104600682s) [seg_pos={0 0}, upload=302109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 9 02:26:27.178428768 (377h14m54.104723465s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:26:27.178455810 (377h14m54.104750501s) [upload=302109]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:26:27.232166184 (377h14m54.158460883s) [upload=302109]: Done uploading segments...
Jul 9 02:26:27.232188444 (377h14m54.158483123s) [upload=302109]: Upload finished. err: <nil>
Duration: 404.557305ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [........X....X.X..XX.XX..X.X.X.....XXX.XX.X...XXX....XX...X.] duration:
7.492267449s
FAIL (2025-07-09 02:44:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.492298348s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
39 out of 60 is failed (threshold=3) [.XXXXXXX..XX....XX.XXX.XXX.XX.XXXXXX.XXXXX.XX.....X..XXXXX.X] duration:
1m0.001568832s
FAIL (2025-07-09 02:44:17 UTC)
operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001587102s
core
uplink
us-select-1
us1-placement-20
|